Search for: All records

Modality fusion is a cornerstone of multimodal learning, enabling information integration from diverse data sources. However, vanilla fusion methods are limited by (1) inability to account for heterogeneous interactions between modalities and (2) lack of interpretability in uncovering the multimodal interactions inherent in the data. To this end, we propose I2MoE (Interpretable Multimodal Interaction-aware Mixture of Experts), an end-to-end MoE framework designed to enhance modality fusion by explicitly modeling diverse multimodal interactions, as well as providing interpretation on a local and global level. First, I2MoE utilizes different interaction experts with weakly supervised interaction losses to learn multimodal interactions in a data-driven way. Second, I2MoE deploys a reweighting model that assigns importance scores for the output of each interaction expert, which offers sample-level and dataset-level interpretation. Extensive evaluation of medical and general multimodal datasets shows that I2MoE is flexible enough to be combined with different fusion techniques, consistently improves task performance, and provides interpretation across various real-world scenarios. 

Recent advancements in large language models (LLMs) have achieved promising performances across various applications. Nonetheless, the ongoing challenge of integrating long-tail knowledge continues to impede the seamless adoption of LLMs in specialized domains. In this work, we introduce DALK, a.k.a. Dynamic Co-Augmentation of LLMs and KG, to address this limitation and demonstrate its ability on studying Alzheimer's Disease (AD), a specialized sub-field in biomedicine and a global health priority. With a synergized framework of LLM and KG mutually enhancing each other, we first leverage LLM to construct an evolving AD-specific knowledge graph (KG) sourced from AD-related scientific literature, and then we utilize a coarse-to-fine sampling method with a novel self-aware knowledge retrieval approach to select appropriate knowledge from the KG to augment LLM inference capabilities. The experimental results, conducted on our constructed AD question answering (ADQA) benchmark, underscore the efficacy of DALK. Additionally, we perform a series of detailed analyses that can offer valuable insights and guidelines for the emerging topic of mutually enhancing KG and LLM. 

Abstract The main subject of this study is the low‐frequency (with the periods longer than 2 hr) wave processes in the coupled regional system of the Ross Ice Shelf (RIS), the Ross Sea and the atmosphere above them. We investigate possible causal relationships between the wave activity in the three media using a unique set of geophysical instruments: a hydrophone measuring pressure variations on the seafloor, a network of seismometers measuring vertical displacements of the RIS surface, and a Dynasonde system measuring wave characteristics at the ionospheric altitudes. We present an extension of the previously introduced theoretical model of the coupled resonance vibrations of the RIS that quantifies the connection between the ocean tide and the resonance vibrations of the RIS. The ocean tide is confirmed as the most significant source of excitation of the resonances. Analysis of average power spectra in year‐long data sets reveals multiple harmonics of the tide (eight) detected by the RIS seismometers while only three are detected by the seafloor sensor. This may represent a confirmation of the effect of resonance‐related broadband amplification predicted by the model. Several peaks in the spectrum of RIS vibrations have periods different from the periods of nearby tidal constituents and may be associated with broad‐scale resonance RIS vibrations. Resonances may play a role in maintaining the coupled atmosphere‐ocean wave activity. Our results reveal a statistically significant correlation between the spectra of the vertical displacements of the RIS and the spectra of the atmospheric waves. 

A part of the Southern Ocean, the Ross Sea, together with the Ross Ice Shelf and the atmosphere over the region represent a coupled system with respect to the low-frequency (with the periods longer than 1 hour) wave processes observed in the three media. We study interconnections between them using a unique combination of geophysical sensors: hydrophones measuring pressure variations on the bottom of the open ocean, seismographs measuring vertical displacements of the surface of the Ross Ice Shelf, and the Jang Bogo Dynasonde system measuring wave parameters at the altitudes of the lower thermosphere. Analysis of a year-long data sets from Ross Ice Shelf-based instruments reveals presence in their average power spectra of the peaks in the 2-11 hours period range that may be associated with the low-order resonance vibrations of the system. More harmonics of the 24 hour tide (seven) are detected by the RIS seismographs compared to the sea floor sensor (where only two are clearly visible). This may be a consequence of the RIS resonance-related broadband amplification effect predicted by our model. There are several peaks in the RIS vibration spectrum (T = 8.37, 8.23, 6.3 and 6.12 hours) that are not detected by the hydrophone and may be directly related to RIS resonances. The prominent T = 25.81 hour peak is a likely candidate for the sub-inertial RIS resonance. The periods of lower RIS resonance modes predicted by our simple model and the observed spectral peaks are in the same general band. This is the first direct observation of the resonance effects in vibrations of the Ross Ice Shelf. Our results demonstrate the key role of the resonances of the Ross Ice Shelf in maintaining the wave activity in the entire coupled system. We suggest that the ocean tide is a major source of excitation of the Ross Ice Shelf’s resonances. The ice shelf vibrations may also be supported by the energy transfer from wind, swell, and infragravity wave energy that couples with the ice shelf. Overlapping 6-month-long data sets reveal a significant linear correlation between the spectra of the vertical shifts of the Ross Ice Shelf and of the thermospheric waves with the periods of about 2.1, 3.7, and 11.1 hours. This result corroborates earlier lidar observations of persistent atmospheric wave activity over McMurdo. We propose a theory that quantifies the nexus between the ocean tide and the resonance vibrations of the Ross Ice Shelf. It complements the theoretical model of the process of generating the atmospheric waves by the resonance vibrations of the Ross Ice Shelf published by us earlier. 

Abstract Gravity waves impacting ice shelves illicit a suite of responses that can affect ice shelf integrity. Broadband seismometers deployed on the Ross Ice Shelf, complemented by a near‐icefront seafloor hydrophone, establish the association of strong icequake activity with ocean gravity wave amplitudes (A_G) below 0.04 Hz. The Ross Ice Shelf‐front seismic vertical displacement amplitudes (A_SV) are well correlated withA_G, allowing estimating the frequency‐dependent transfer function from gravity wave amplitude to icefront vertical displacement amplitude (T_GSV(f)).T_GSV(f) is 0.6–0.7 at 0.001–0.01 Hz but decreases rapidly at higher frequencies. Seismicity of strong icequakes exhibits spatial and seasonal associations with different gravity wave frequency bands, with the strongest icequakes observed at the icefront primarily during the austral summer when sea ice is minimal and swell impacts are strongest.